Prepositional Phrase Attachment without Oracles

Work on PP attachment resolution generally assumes that there is an oracle that provides the two hypothesized structures that we want to choose between. The information that there are two possible attachment sites and the information about the lexical heads of those phrases is usually extracted from gold standard parse trees. We show that the performance of reattachment methods is higher with such an oracle than without. Since oracles are not available in NLP applications, this indicates that the current evaluation methodology for PP attachment does not produce realistic performance numbers. We argue, that PP attachment should not be evaluated in isolation, but instead as an integral component of a parsing system, without using information from the gold-standard oracle.